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Abstract 

Naval  Air  Warfare  Center-Aircraft  Division  (NAWCAD)  wants  to  ensure  that  program  offices 
derive  the  best  value  for  their  resources.  These  program  offices  need  to  understand  their 
most  effective  cost,  schedule,  and  performance-based  solutions  when  faced  with  an 
acquisition  requirement.  Comparisons  for  sourcing  are  complex  and  require  extensive 
decision  support  analysis  to  understand  the  solution  space  and  foresight  between  original 
equipment  manufacturers  (OEMs)  and  government  entities  performing  the  Lead  System 
Integrator  (LSI)  role.  The  research  focused  on  the  question,  how  can  program  managers  best 
determine  their  acquisition  source  solutions  in  a  timely  and  cost-effective  manner? 

The  method  used  was  to  develop  a  simulation  tool  to  assist  program  offices  in  evaluating  the 
relative  risks  of  utilizing  NAWCAD  and  commercial  OEMs  for  various  LSI  roles  and 
responsibilities.  Lone  Star  Aerospace  Inc.  was  tasked  to  develop  a  simulation  tool  to  address 
highly  complex  issues  for  customers  in  NAVAIR. 

Eleven  NAVAIR  program  managers  (PMs)  and  16  competency  leads  were  interviewed,  and 
other  stakeholders  were  surveyed  online.  Risk  drivers  were  codified  into  a  model  based  on 
lessons  learned  from  PMs’  experience.  A  prototype  LSI  directional  tool  was  developed  and 
demonstrated.  Four  beta  tests  were  conducted  on  NAVAIR  programs  to  stress  the  model’s 
capability  and  to  identify  potential  improvements. 

Introduction 

Signed  January  7,  2015,  Department  of  Defense  Instruction  (DoDI)  5000.2  states 
under  Enclosure  2:  Program  Management, 
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The  Program  Manager  will  develop  and  execute  an  approved  Acquisition 
Strategy.  This  document  is  the  Program  Manager’s  plan  for  program 
execution  across  the  entire  program  life  cycle.  It  is  a  comprehensive, 
integrated  plan  that  identifies  the  acquisition  approach  and  key  framing 
assumptions,  and  describes  the  business,  technical,  and  support  strategies 
that  the  Program  Manager  plans  to  employ  to  manage  program  risks  and 
meet  program  objectives.  The  strategy  must  reflect  the  Program  Manager’s 
understanding  of  the  business  environment;  technical  alternatives;  small 
business  strategy;  costs,  risks  and  risk  mitigation  approach.  The  plan 
supports  successful  delivery  of  the  capability  at  an  affordable  life-cycle  price, 
on  a  realistic  schedule. 

The  business  approach  detailed  in  the  Acquisition  Strategy  should  be 
designed  to  manage  the  risks  associated  with  the  product  being  acquired.  It 
should  fairly  allocate  risk  between  industry  and  the  government.  The 
approach  will  be  based  on  a  thorough  understanding  of  the  risks  associated 
with  the  product  being  acquired  and  the  steps  that  should  be  taken  to  reduce 
and  manage  that  risk.  The  business  approach  should  be  based  on  market 
analysis  that  considers  market  capabilities  and  limitations.  The  contract  type 
and  incentive  structure  should  be  tailored  to  the  program  and  designed  to 
motivate  industry  to  perform  in  a  manner  that  rewards  achievement  of  the 
government’s  goals. 

This  development  of  an  Acquisition  Strategy  by  the  PM  has  become  increasingly 
complicated  by  operating  in  a  VUCA  environment.  VUCA  stands  for  volatility,  uncertainty, 
complexity,  and  ambiguity.  A  growing  body  of  research  on  VUCA — including  the  2014 
dissertation  of  Dale  Moore,  Director  of  the  Naval  Air  Warfare  Center-Aircraft  Division 
(NAWCAD)  Strategic  Cell — has  recognized  that  a  VUCA  environment  triggers  more  frequent 
strategic  reorientations.  When  strategies  change  rapidly,  organizations  can  flounder  and 
resources  can  be  wasted.  When  strategies  do  not  adapt  to  the  VUCA  environment,  they  can 
become  obsolete  quickly,  also  triggering  wasteful  spending  and  suboptimal  acquisition 
strategy,  particularly  for  traditional  program  development  lead  times. 

At  the  Defense  Acquisition  University  Small  Business  Hot  Topic  Forum  on  May  8, 
2013,  Nickolas  H.  Guertin,  PE  DASN  RDT&E,  gave  a  brief  on  open  systems  architecture. 
The  key  message  was  that  the  legacy  business  model  is  one  big  program  that  equals  one 
big  contract  and  that  a  future  business  model  will  invoke  competition  across  the  life  cycle 
using  multiple  acquisition  sources. 

The  NAVAIR  Long-Range  Strategy  (LRS)  identified  government  as  Lead  Systems 
Integrator  (LSI)  as  a  future  desired  state  (Naval  Air  Systems  Command  [NAVAIR],  2013).  It 
involves  development  and  management  of  government-owned  capability,  based  on 
technical  standards  and  re-use  of  engineering  and  test  results  associated  with  those 
standards  to  reduce  acquisition  cost. 

In  Better  Buying  Power  3.0  (BBP  3.0),  Frank  Kendall  asserts,  “Nothing  is  more 
important  to  our  success  than  our  professional  ability  to  understand,  think  critically,  and 
make  sound  decisions  about  the  complex  and  often  highly  technical  matters  defense 
acquisition  confronts”  (Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics  [OUSD(AT&L)j,  2014),  Underpinning  BBP  3.0  is  the  growing  concern  that  the 
United  States’  technological  superiority  over  potential  adversaries  is  being  threatened  today 
in  a  way  that  we  have  not  seen  for  decades. 
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This  new  and  focused  BBP  3.0  initiative  will  strengthen  the  Department’s  organic 
military  and  government  civilian  technical  expertise.  The  Department  cannot  be  an  effective 
customer  for  technical  excellence  and  innovation  if  we  do  not  embody  those  characteristics 
fully  in  our  own  workforce.  We  cannot  make  decisions  about  technology  if  we  don’t  fully 
understand  what  is  possible  and  how  to  achieve  it. 

In  an  ITEA  Journal  article,  “DoD  Labs,  Navy  Warfare  Centers  are  Untapped 
Resources,”  RDML  Michael  Moran  and  SES  Scott  O'Neil  discuss  the  89%  industry  activity 
and  the  11%  government  (Navy  data)  and  how  that  needs  to  be  re-examined  along  with 
considering  more  LSI  work  and  leveraging  in-house  assets.  Currently,  the  government 
workforce  is  not  fully  positioned  to  assume  the  LSI  role  on  a  broad  scale,  but  if  we  engage 
early  in  the  technical  development  efforts,  whenever  it  makes  sense,  we  can  build  a  robust 
capability.  Although  this  paper  is  specifically  focused  on  experiences  at  Naval  Air  Warfare 
Center  Weapons  Division,  there  is  anecdotal  evidence  showing  that  these  issues  are  typical 
of  many  Navy/DoD  warfare  centers  and  laboratories. 

Purpose  and  Research  Question 

The  purpose  of  this  qualitative  research  study  is  twofold.  First  is  to  provide  PMs  with 
a  simulation  tool  to  assist  program  offices  in  evaluating  the  relative  merits  and  risks  of 
utilizing  NAWCAD  and  commercial  OEMs  for  various  LSI/LCI  roles  and  responsibilities. 
Second  is  to  understand  where  the  government  (NAWCAD)  lacks  competitiveness  on  the 
capability  to  perform  LSI  (identify  the  gaps).  The  research  focused  on  the  question,  how  can 
program  managers  best  determine  their  acquisition  source  solutions  in  a  timely  and  cost- 
effective  manner? 

NAWCAD  wants  to  ensure  that  the  program  offices  that  it  supports  derive  the  best 
value  for  their  resources.  These  program  offices  need  to  understand  their  most  effective 
cost,  schedule,  and  performance-based  solutions  when  faced  with  an  acquisition 
requirement.  Comparisons  for  sourcing  are  complex  and  require  extensive  decision  support 
analysis  to  understand  the  solution  space  and  direct  more  detailed  cost  estimates  between 
OEMs  and  government  entities  performing  the  LSI  role  or  Lead  Capabilities  Integration  (LCI) 
role. 

Methodology 

Extensive  market  research  was  conducted  with  leading  edge  technology,  best 
practices,  and  analytical  competitors  like  Lone  Star  Aerospace  Inc.,  IBM,  Decision  Lens, 
Defense  &  Security  Technology  Group  Inc.,  and  Deloitte’s  Highly  Immersive  Visual 
Environment  (HIVE).  Additionally,  a  literature  review  was  conducted  to  research  state  of  the 
art  publications,  articles,  and  industry  Communities  of  Interest  (COI).  Lone  Star  Aerospace 
Inc.  was  tasked  to  develop  an  LSI  directional  tool  to  assess  Courses  of  Action  (CoA)  and 
provide  relative  risks  of  utilizing  NAWCAD  and  OEMs. 

The  use  of  an  appropriate  decision  process  is  important  to  provide  best  outcomes. 
During  the  execution  of  this  research,  the  best  practice  of  the  Dialogue  Decision  Process 
from  the  Handbook  of  Decision  Analysis  was  used  (Parnell  et  al. ,  2013,  Section  5.3).  The 
dialogue  decision  process  is  a  decision  process  that  focuses  on  a  structured  dialogue 
between  the  Decision  Board  and  Project  Team.  This  was  accomplished  through  several 
meetings  that  included  a  kick-off  meeting,  interim  review  meeting,  LSI  Directional  Tool 
Demo/Feedback  meeting,  and  the  beta  test  out-brief. 

A  kick-off  meeting  for  the  LSI  decision  model  was  held  October  24,  2013,  with 
NAWCAD  leadership,  stakeholders,  and  subject  matter  experts  (SMEs).  The  purpose  was  to 
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reach  consensus  on  the  model  requirements,  to  establish  support,  synchronize,  and  provide 
guidance  on  alternatives  and  project  strategy. 

Model  Requirements 

The  directional  model  needed  to  be  able  to  assess  roles,  responsibilities,  and 
functions  of  key  entities  and  the  elements  that  support  those  functions.  The  model  also 
needed  to  be  able  to  differentiate  between  two  or  more  alternatives  and  be  used  across  the 
entire  life  cycle  or  a  portion  of  a  program.  This  discrimination  was  a  prelude  to  detailed 
planning  which  followed  the  analysis  conducted  with  this  model.  The  minimum  areas  for  the 
model  to  assess  in  each  Course  of  Action  (CoA)  were  cost,  schedule,  technical 
performance,  capability,  and  staffing  risks.  It  also  needed  to  be  scalable  to  support  all 
program  types,  as  well  as  be  validated  on  beta  project(s). 

Specifically,  tools,  processes,  and  analysis  needed  to  provide  insight  to  efforts  being 
planned  or  performed,  with  emphasis  on  cost,  time,  labor  mix,  risk,  and  performance  as  key 
measures.  The  tool  needed  to  be  designed  as  a  decision  aid  and  was  not  meant  to  supplant 
or  replace  Program  Executive  Office  (PEO),  Program  Management  Air  (PMA),  or  NAVAIR 
support  systems. 

An  LSI  decision  model  preliminary  interim  review  meeting  was  held  November  13, 
2013.  The  preliminary  interim  review  provided  SME  feedback  on  the  proposed  LSI  decision 
model  architecture,  synchronized  expectations,  and  provided  guidance  on  next  steps. 
Attendees  conducted  a  review  of  the  developing  government  LSI  Decision  Tool.  The  high 
level  approach  included  Course  of  Actions  (CoAs)  based  on  MIL-STD-881  Work  Breakdown 
Structure  (WBS)  level  2/3  and  establishes  a  technical  baseline  that  is  consistent  across  all 
CoAs.  High  level  model  architecture  included  Control  Panels,  Baseline  WBS  Lv  2/3,  Revised 
Baseline  WBS  Lv  2/3,  Contractor  Capabilities,  Government  Capabilities,  and  Decision 
Metrics. 

SME  Interviews 

The  next  step  in  the  process  was  to  conduct  SME  interviews.  This  step  provided 
insight  into  the  merit  of  approach  and  architecture.  The  support  and  feedback  received 
previous  insights  that  were  placed  into  a  model  influence  diagram  that  was  then  developed 
into  a  mathematical  structure  with  98%  of  the  math  completed  and  the  remaining  notional 
data  loaded  into  the  demonstration. 

Eleven  NAVAIR  deputy  program  managers  and  16  competency  leads  were 
interviewed  and  other  stakeholders  were  surveyed  online.  The  results  were  used  to  codify 
risk  drivers  into  the  model  based  on  lessons  learned  from  PMs’  experience.  Program 
managers,  deputy  program  managers,  and  competency  leads  were  selected  with  the 
following  criteria:  (1)  performed  in  a  senior  acquisition  program  management  role,  to  include 
program  manager  or  deputy  program  manager  positions  for  over  two  years;  (2)  operated  at 
a  senior  level  of  the  organization  (military  0-6/civilian  GS-15,  or  equivalent)  with  significant 
responsibility;  and  (3)  was  involved  with  and  responsible  for  developing  and  conceptualizing 
long-term  acquisition  program  strategies  and  plans.  Competency  leads  were  selected  for 
their  technical  expertise  and  knowledge. 

Data  was  collected  through  30-minute  interviews  with  PMs  and  60-minute  interviews 
with  competency  leads.  The  interview  sought  to  understand  PMA  perspectives  and 
experiences  in  assessing  entity  role  selection  when  planning  and  executing  a  project  or 
program.  An  “entity”  is  an  organization  (industry  or  government)  performing  a  significant  role 
in  a  system  or  program.  The  interviews  were  conducted  using  the  identified  best  practices  in 
the  Handbook  of  Decision  Analysis  as  a  guide  (Parnell  et  al.,  2013,  Section  4.6). 
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Interview  Questions 

The  interview  questions  were  as  follows: 

1 .  When  structuring  the  roles  and  responsibilities  of  various  entities  (across 
industry  and  government)  in  a  program,  what  is  your  vision  and/or  objective 
when  determining  who  has  which  roles  and  responsibilities? 

2.  We  are  tasked  to  look  at  Cost,  Schedule,  and  Performance  as  criteria. 

a.  What  aspects  of  these  three  do  you  believe  are  the  most  important? 

b.  What  other  criteria  do  you  feel  are  important  when  assessing  these 
roles  and  responsibilities? 

3.  Can  you  share  an  example  of  where  an  allocation  of  roles  and 
responsibilities  among  entities  has  been  defined  and  implemented  well? 

4.  How  was  the  lead  role  distributed  among  these  entities? 

5.  How  big  of  a  factor  are  the  “values”  of  the  entity  to  you  when  determining  the 
structure  of  the  program  as  it  relates  to  the  roles  and  responsibilities? 

6.  What  issues  define  compatibility  among  the  entities  that  must  collaborate  for 
a  successful  program?  In  particular,  what  enables  two  or  more  entities  to 
successfully  create  a  systems  capability? 

7.  Can  you  share  an  example  of  where  an  allocation  of  roles  and 
responsibilities  among  entities  has  not  been  defined  and  implemented  well? 

8.  What  metrics  are  you  held  accountable  to  when  executing  a  program? 

a.  What  is  the  source  of  the  metric  requirement? 

b.  Where  does  the  data  for  those  metrics  reside? 

c.  Which  of  these  metrics  are  most  useful  to  you  in  executing  your 
program? 

d.  Which  of  these  metrics  provides  any  insight  to  the  successful 
allocation  of  roles  and  responsibilities  of  the  entities  on  your  program? 

Prototype  LSI  Directional  Tool 

An  LSI  Directional  Tool  Demo/Feedback  meeting  was  held  February  5,  2014. 
Attendees  were  both  PMAs  and  NAWCAD  leadership.  The  prototype  LSI  directional  tool 
was  developed  to  demonstrate  user  input  and  output  formats.  The  purpose  of  the 
demonstration  was  to  gather  the  voice  of  the  customer  for  feedback,  recommended 
changes,  and  improvements,  and  to  synchronize  expectations  before  locking  into  a  final 
configuration  and  to  identify  the  PMAs’  desire  to  conduct  beta  tests. 

Although  the  demonstrated  model  had  notional  data  loaded,  it  demonstrated  the 
effectiveness  of  the  tool.  Lone  Star  conducted  surveys  of  the  PMA,  and  competency  and 
subject  matter  experts  throughout  industry  to  quantify  the  real  world  impacts  to  cost, 
schedule,  and  performance  based  on  capability  and  program  assessments.  The  survey  data 
represented  a  quantified  representation  of  the  real  world  experience  of  program  managers, 
competency  leads,  and  industry  SMEs.  The  information  received  subsequently  replaced  the 
notional  data  utilized  prior  to  beta  testing.  Additionally,  it  was  pointed  out  that  NAWCAD  is 
pursuing  21st  century  methodologies  and  tools  to  perform  its  mission  in  the  future.  These 
include  product  lifecycle  management  tools,  digital  engineering,  cloud  computing,  data 
fusion,  model-based  research,  and  engineering,  prototyping,  and  agile  systems  engineering, 
in  which  historical  data  does  not  exist. 
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The  prototype  allows  a  program  manager  to  establish  a  Baseline  CoA  based  on  the 
program’s  level  2/3  WBS  for  cost,  schedule,  and  performance. 

The  initial  CoA  can  be  anywhere  on  the  continuum  provided  in  Figure  1.  The  PM  or 
Integrated  Program  Team  (IPT)  can  evaluate  potential  CoAs  for  any  combination  of 
government  or  industry  LSI  activity  at  the  system,  sub-system,  or  component  level. 


Figure  1.  Courses  of  Action  on  a  Continuum 

Once  the  inputs  are  made  for  the  Baseline  CoA,  the  program  IPT  completes  an 
assessment  of  the  Baseline  CoA  in  the  following  areas  that  drive  risk: 

•  Technical  complexity  (TRL)  and  component  or  software  re-use 

•  System  complexity 

•  Program  complexity 

•  Contract  type 

•  Contract  award  time 

•  Integration  readiness 

•  Proprietary  data  rights 

•  Competency  of  personnel 

•  Availability  of  personnel 

•  Facility  readiness 

•  Security  clearance  status 

•  Pass  through  cost 

These  assessments  result  in  a  risk  assessment  of  the  Baseline  CoA  relative  to  the 
Baseline  Inputs.  Subsequent  CoAs  can  then  be  assessed  based  on  the  reallocation  of  work 
between  OEMs,  the  government,  or  another  OEM.  These  CoAs  can  then  be  compared  to 
each  other  to  determine  the  relative  risk  of  each  CoA,  as  well  as  the  most  feasible  course  of 
action  to  take. 

Risk  Assessment  Descriptions 

Technical  Complexity  (TRL)  and  Component  or  Software  Re-Use 

The  Technology  Readiness  Assessment  (TRA)  framework  was  developed  to  assess 
the  maturity  of  technology  elements  via  the  established  metrics  of  Technology  Readiness 
Levels  (TRLs).  The  TRL  metric  assigns  a  ranking  to  the  maturity  of  the  technology  element 
using  specific  evaluation  criteria;  the  scale  can  be  used  to  assess  the  risk  of  developing  and 
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transitioning  new  technologies  in  the  context  of  a  larger  system  acquisition.  The  framework 
of  Re-Use  is  similar  to  TRL  and  assesses  the  percentage  of  re-use  for  any  given  WBS 
development  effort. 

TRL  Levels 

•  TRL  1 — Basic  principles  observed  and  reported 

•  TRL  2 — Technology  concept  and/or  application  formulated 

•  TRL  3 — Analytical  and  experimental  critical  function  and/or  characteristic 
proof-of  concept 

•  TRL  4 — Component/subsystem  validation  in  laboratory  environment 

•  TRL  5 — System/subsystem/component  validation  in  relevant  environment 

•  TRL  6 — System/subsystem  model  or  prototyping  demonstration  in  a  relevant 
end-to-end  environment  (ground  or  space) 

•  TRL  7 — System  prototyping  demonstration  in  an  operational  environment 
(ground  or  space) 

•  TRL  8 — Actual  system  completed  and  “mission  qualified”  through  test  and 
demonstration  in  an  operational  environment  (ground  or  space) 

•  TRL  9 — Actual  system  “mission  proven”  through  successful  mission 
operations  (ground  or  space) 

Re-Use  Levels 

1  =  Less  than  10% 

2  =  10-%  -20% 

3  =  20%  -  30% 

4  =  30%  -  40% 

5  =  40%  -  50% 

6  =  50%  -  60% 

7  =  60%  -  70% 

8  =  70%  -  80% 

9  =  80%  -  90% 

10  =  Greater  than  90% 

Once  the  detail  hardware/software  elements  of  each  WBS  are  assessed,  a  System 
TRL  and  System  Re-Use  are  computed  based  on  weighting  those  elements  based  on  cost. 
The  derived  System  level  TRL  and  Re-Use  has  proven  effective  in  determining  the  level  of 
risk  to  cost  and  schedule  of  a  given  CoA. 

System  Complexity 

The  work  indicated  that  while  TRL  and  Re-Use  are  important  criteria  and  both  should 
be  considered  and  can  be  successfully  modeled,  a  reliable  method  for  categorizing  system 
complexity  to  a  usable  scale  for  programs  and  products  in  government  and  industry  is 
needed  for  objective  planning  and  analysis. 

System  complexity  has  two  dimensions: 

•  Dimension  1 — Product  &  Context  Complexity  (Functions) 
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Lone  Star  Aerospace  Inc.  adapted  research  results  from  the  MIT  Sloan  School  that 
provide  a  relative  complexity  score  for  functions  (Product  and  Context  Complexity).  The 
PMA  assesses  the  complexity  of  the  function  of  the  system  based  on  the  following  criteria: 

1  =  A  part  or  component  with  a  single  function  and  one  interface  type 

2  =  A  sub-assembly  or  mechanism,  composed  of  #1  s,  with  one  or  a  few 

functions,  and  few  interface  types  (these  can  be  nested — sub-assemblies 
can  merge  to  create  more  complex  things  that  are  still  rated  2);  a  circuit 
board  is  a  2. 

3  =  A  machine,  apparatus,  or  functional  item,  composed  of  #2’s  which  may 

have  multiple,  complex  functions,  and  may  have  many  interface  types. 
Pacemakers,  server  blades  and  laser  guided  bombs  are  all  3’s. 

4  =  A  system  or  complex  apparatus  composed  of  3’s  with  highly  complex 

functionally  and  diverse  interface  types.  Communications  satellites,  air 
traffic  control  radars,  airliners,  telecom  switches,  and  core  network  routers 
are  all  examples  of  4’s. 

5  =  A  system  of  systems,  composed  of  4’s  (and  lower  complexity  products).  A 

telecom  network,  major  airport,  and  carrier  battle  group  are  examples  of 
5’s. 

•  Dimension  2 — Interface  Complexity  &  Information  Diversity 

For  Interface  Complexity/Information  Diversity,  the  PMA  will  make  an  assessment  on 
the  complexity  based  on  the  following  criteria: 

1  =  An  interface  of  a  single  type,  single  purpose,  and  low  information  diversity 

2  =  An  interface  of  1  to  3  types,  1  to  3  purposes,  and  with  low  information 

diversity 

3  =  An  interface  more  complex  than  #2  in  only  one  or  two  descriptors 

4  =  An  interface  exceeding  the  complexity  of  #2  in  all  three  descriptors 

5  =  An  interface  managing  many  #4’s 

When  system  complexity  attributes  are  accounted  for  along  with  TRL  and  Re-Use,  it 
provides  program  managers  the  ability  to  better  plan  for  cost  and  schedule  risk  in  their 
program  as  they  develop  program  plans. 

Program  Complexity 

Based  on  feedback  through  the  interview  process,  the  researchers  determined  that 
Program  Complexity  had  two  dimensions,  namely  customer  complexity  and  supplier 
complexity.  As  each  CoA  is  defined,  the  resultant  structure  of  the  program  changes  based 
on  the  number  of  pieces  that  might  need  to  be  managed.  Lone  Star  developed  a 
methodology  to  determine  the  impact  to  changes  in  the  complexity  of  managing  the  program 
based  on  the  CoA  being  evaluated. 

Customer  complexity  consists  of  three  components: 

•  The  number  and  type  of  customers  ranging  from  a  single  DoD  customer  to 

multiple  DoD  and  international  customers 

•  The  number  of  contracts  or  contracting  documents  from  that  customer  to  be 

managed 
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•  Whether  the  program  is  a  development  effort  or  is  in  production 

Supplier  complexity  consists  of  two  components: 

•  The  types  of  supplier  contracts  to  be  managed,  ranging  from  all  “Build  to 
Print”  to  all  “Development”  contracts 

•  The  relationship  with  and  between  suppliers 

Different  CoAs  will  result  in  numbers  of  customers/suppliers,  as  well  as  the  nature  of 
the  relationships  required  to  execute  a  program.  By  accounting  for  the  impact  of  these 
differences,  the  PM  is  better  able  to  understand  and  manage  the  potential  impacts  to  cost 
and  schedule  of  any  CoA. 

Contract  Type 

A  relationship  between  Contract  Type  and  Contract  Cost  were  established  based  on 
prior  Lone  Star  research.  The  ability  to  execute  within  budget  is  impacted  depending  on  the 
contract  type  utilized  on  a  subcontract.  Five  contract  types  that  are  modeled  include  the 
following: 

•  Firm  Fixed  Price 

•  Firm  Fixed  Price  w /  Simplistic  Incentive  Fee 

•  Firm  Fixed  Price  w /  PMF  Incentive  Fee,  Incentives,  and  Process  Authority 

•  Cost  Plus  Fixed  Fee 

•  Cost  Plus  w /  PMF  Incentive  Fee,  Incentives,  and  Process  Authority 

Different  CoAs  will  result  in  different  quantities  and  sizes  of  contracts  to  be  issued 
and  managed.  One  CoA  could  have  one  large  prime  contract,  while  another  could  have 
multiple  smaller  contracts.  While  smaller  contracts  can  result  in  smaller  contracting  cycle 
time,  the  people  required  to  support  multiple  awards  may  or  may  not  be  available. 
Additionally,  the  PMA  may  or  may  not  have  sufficient  insight  or  knowledge  about  the  supply 
chain  to  support  multiple  awards. 

Contract  Award  Time 

Based  on  planning  factors  from  Procurement  Administrative  Lead  Time  (PALT),  the 
researchers  modeled  an  estimated  cycle  time  for  contracting  to  be  measured  against  a 
planned  baseline.  Historical  contract  cycle  time  is  based  on  the  following: 

•  Contract  PALT  # 

•  Contract  PALT  Type 

•  Contract  Competitive/Any  Value 

•  Contract  Defined  Price 

•  Contract  Re-Use 

•  Contracting  Resources 

This  provides  the  PM  with  insight  to  the  cycle  time  required  to  get  any  given  CoA 
under  contract  to  begin  execution. 

Integration  Readiness 

Integration  Readiness  was  based  on  Integration  Readiness  Criteria  from  AIR  4.1  and 
an  Integration  White  Paper  from  Jennifer  Long  (2011).  While  TRL  and  Re-Use  are  important 
criteria  and  provide  insight  to  the  relative  risk  of  a  set  of  CoAs,  the  ability  to  integrate 
hardware  and  software  together  can  pose  significant  risk  to  program  execution  if  not 
accounted  for  in  the  evaluation  of  each  CoA. 
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An  Integration  Readiness  Level  (IRL)  assessment  was  developed  that  relates  the 
“integration  characteristics”  of  configuration  items  to  integration  risk  levels.  The  assessment 
was  structured  along  the  same  lines  as  the  system  complexity  assessment,  but  was 
intended  to  allow  assessment  of  the  risk  associated  with  software  and  hardware 
development  of  configuration  items  (CIs). 

Proprietary  Data  Rights 

DoDI  5000.2  (DoD,  2015)  states  under  Enclosure  2:  Program  Management, 

(4)  Intellectual  Property  (IP)  Strategy  and  Open  Systems  Architectures. 

Program  management  must  establish  and  maintain  an  IP  Strategy  to  identify 
and  manage  the  full  spectrum  of  IP  and  related  issues  (e.g.,  technical  data 
and  computer  software  deliverables,  patented  technologies,  and  appropriate 
license  rights)  from  the  inception  of  a  program  and  throughout  the  life  cycle. 

The  IP  Strategy  will  describe,  at  a  minimum,  how  program 
management  will  assess  program  needs  for,  and  acquire  competitively 
whenever  possible,  the  IP  deliverables  and  associated  license  rights 
necessary  for  competitive  and  affordable  acquisition  and  sustainment  over 
the  entire  product  life  cycle,  including  by  integrating,  for  all  systems,  the  IP 
planning  elements  required  by  subpart  207.106  (S-70)  of  the  Defense 
Federal  Acquisition  Regulation  Supplement  (Reference  (al))  for  major 
weapon  systems  and  subsystems  thereof. 

The  IP  Strategy  will  be  updated  throughout  the  entire  product  life 
cycle,  initially  as  part  of  the  Acquisition  Strategy,  and  during  the  Operations 
and  Support  Phase  as  part  of  the  Life-Cycle  Sustainment  Plan. 

Program  management  is  also  responsible  for  evaluating  and 
implementing  open  systems  architectures,  where  cost  effective,  and 
implementing  a  consistent  IP  Strategy.  This  approach  integrates  technical 
requirements  with  contracting  mechanisms  and  legal  considerations  to 
support  continuous  availability  of  multiple  competitive  alternatives  throughout 
the  product  life  cycle. 

Competency  and  Availability  of  Personnel 

People  Readiness  was  characterized  as  having  two  components,  including 
personnel  competency  and  availability.  The  PMA  is  required  to  make  an  assessment  to 
determine  whether  an  executing  entity  has  the  competency  and  availability  to  perform  the 
expected  work  by  utilizing  a  Likert  assessment  scale  calculated  as  follows. 


•  Competency: 

1  =  No  competency,  sourcing  can  be  obtained 

2  =  Some  competency  with  major  additions  needed 

3  =  Most  competencies  with  few  additions  needed 

4  =  Most  competencies  with  minor  additions 

5  =  All  competencies  are  capable 

•  Availability: 

1  =  None  available,  sourcing  can  be  obtained 

2  =  Some  available  with  major  additions  needed 

3  =  Most  available  with  few  additions  needed 
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4  =  Most  available  with  minor  additions 

5  =  All  personnel  are  available 


Facility  Readiness 

Facilities  Readiness  uses  a  Likert  scale  assessment  of  the  facilities  available  to  do 
the  work,  including  Security  readiness.  The  PMA  will  select  whether  an  executing  entity  has 

1  =  None  available,  sourcing  can  be  obtained 

2  =  Some  available  with  major  additions  needed 

3  =  Most  available  with  few  additions  needed 

4  =  Most  available  with  minor  additions 

5  =  All  facilities  available 

Security  Clearance  Status 

Clearance  Availability  is  a  Likert  scale  assessment  of  the  availability  of  cleared 
people  to  do  the  work.  The  PMA  will  be  able  to  select  whether  an  executing  entity  has 

1  =  None  available,  sourcing  can  be  obtained 

2  =  Some  available  with  major  additions  needed 

3  =  Most  available  with  few  additions  needed 

4  =  Most  available  with  minor  additions 

5  =  All  cleared  people  available 

By  assessing  the  People,  Facilities,  and  Clearance  Availability  of  each  executing 
entity  of  a  given  CoA,  the  program  manager  has  the  insight  to  the  resources  available  to 
execute  that  CoA.  Resource  availability  has  a  significant  impact  on  the  ability  to  meet 
schedule  and  ultimately  cost.  This  assessment  not  only  provides  insight  to  schedule  and 
cost  risk  to  the  program  manager,  the  aggregation  of  this  data  provides  NAWCAD  with 
insight  to  resource  areas  that  need  to  be  developed  or  improved. 

Pass  Through  Cost 

Significant  program  savings  may  be  realized  when  the  government  performs  as  a 
competitive,  qualified  LSI.  AIR  4.2  Should  Cost  data  shows  that  the  representative  supply 
chain  profit  is  a  substantial  portion  of  the  total  weapon  system’s  cost.  This  assumes  a  four 
tier  supply  chain,  in  which  both  profits  decrease  in  lower  tiers,  and  the  percentage  of 
outsourced  materials  decrease  in  lower  tiers.  There  is  an  opportunity  to  assess  all  pass 
through  rates  using  historical  data  across  NAVAIR  programs.  This  provides  options  for 
consideration  for  PMA  acquisition  decisions. 

Allowing  for  Potential  Variation  in  Impact  Areas 

Since  the  inputs,  assessments,  and  impacts  to  those  assessments  are  not 
deterministic,  the  variation  in  inputs  and  assessment  impacts  are  accounted  for  in  the 
modeling  and  simulation  process. 


Node 

Layer 

Input  Node  10 

Input  Node  SO 

Input  Node  90 

Input  Node  LB 

Input  Node  H8 

Govt  Contracting  Plan  Duration  Pro  Election  Current  Phase 

Baseline 

0 

0 

0 

0 

0 

Govt  Contracting  Plan  Duration  Pre Election  Future  Phases 

Baseline 

0 

0 

0 

0 

0 

Figure  2.  Lead  Time  Required  to  Put  Contract  Vehicle  in  Place 

The  variation  in  potential  data  inputs  to  the  model  is  captured  by  50,000  Monte  Carlo 
runs  pulling  a  distribution  of  the  data.  This  is  called  10-base-90  format  for  the  data.  The 
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example  given  here  in  Figure  2  is  for  the  lead  time  required  to  put  a  contract  vehicle  in 
place.  During  team  assessments,  the  facilitator  will  ask  the  following  questions: 

•  What  is  the  minimum  amount  of  time  it  could  take  to  get  on  contract? 

•  What  is  the  maximum  amount  of  time  it  could  take  to  get  on  contract? 

•  10%  of  the  time,  how  bad  could  it  be? 

•  10%  of  the  time,  how  good  could  it  be? 

•  What  is  a  nominal  realistic  expectation  of  contract  award  timeline? 

This  creates  dialogue  and  drives  consensus  on  the  appropriate  distribution  recorded 
into  the  model.  The  10-base-90  format  for  the  data  is  available  for  population  across  all  risk 
drivers  in  the  model.  If  required,  the  assessment  team  can  reach  back  to  the  potential 
executing  entities  for  insight  on  the  risk  driver  in  question. 

This  Monte-Carlo  approach  provides  a  distribution  of  potential  LSI  CoAs  for  the  PM, 
evaluated  from  relative  cost  risk,  schedule  risk,  and  performance  risk  of  potential  outcomes. 
The  relative  risk  curve  is  made  up  of  50,000  points  representing  50,000  potential  outcomes 
for  a  CoA. 

These  assessments  result  in  a  risk  assessment  of  the  Baseline  CoA  relative  to  the 
Baseline  Inputs.  Subsequent  CoAs  can  then  be  assessed  based  on  the  reallocation  of  work 
between  OEMs,  the  government,  or  another  OEM.  These  CoAs  can  then  be  compared  to 
each  other  to  determine  the  relative  risk  of  each  CoA,  as  well  as  the  most  feasible  course  of 
action  to  take. 

The  prototype  LSI  directional  tool  was  determined  to  be  ready  for  beta  testing.  The 
beta  testing  included  four  programs  of  record  that  resulted  in  positive  dialogue  and 
participation. 

Beta  Project 

Beta  Test  Objectives 

Beta  test  objectives  were  designed  to  demonstrate  the  following: 


•  Model  functionality  in  an  operational  environment  with  the  Program 
Management  office  in  the  acquisition  process 

•  Incorporate  lessons  learned  from  the  beta  programs  into  architectural 
changes  to  the  model 

•  Identify  and  modify  model  reporting  and  graphics  required  for  PMA  use  in  the 
acquisition  process 

•  Demonstrate  directional  merits  of  various  courses  of  action  regarding  roles 
and  responsibilities  of  executing  entities  enabling  improved  decision-making 
by  NAVAIR  leadership  and  PMs 


Beta  Criteria 

The  optimal  beta  programs  selection  was  for  the  LSI  model  to  test  a  diverse  cross 
section  of  NAVAIR  types  of  programs  to  exercise  the  different  modules  within  the  model  and 
push  the  boundaries  of  the  model’s  capability  to  identify  areas  for  potential  architectural 
improvement.  The  beta  programs  varied  the  degrees  of  technical,  program,  and  system 
complexity,  and  considered  different  current  and  future  program  phases.  The  intent  was  to 
thoroughly  explore  cost,  schedule,  and  performance  questions  as  they  evaluated  CoAs  and 
be  scoped  to  fit  into  a  six-week  evaluation  period. 
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Beta  Programs 

Optimal  beta  programs  were  selected  based  on  a  cross  section  of  NAVAIR 
programs.  They  exercised  different  modules  within  the  model  and  stressed  the  model’s 
capability  to  identify  areas  for  potential  improvement.  The  beta  programs  had  varying 
technical,  program,  and  system  complexity.  Four  programs  of  record  were  selected. 

Beta  Test  Out-Brief 

During  beta  testing,  the  model  demonstrated  the  ability  to  rapidly  change  the  WBS  to 
be  tailored  by  each  program,  and  ran  several  CoAs  running  per  day  with  each  PMA.  The 
results  of  the  model  runs  were  both  informative  and  validated  the  survey  data.  The  overall 
PMA  responses  were  positive  and  it  was  recognized  that  the  tool  has  application  beyond 
LSI  decisions.  Beta  test  feedback  indicated  its  usefulness  for  programmatic  “what-if’ 
analysis  and  other  PMA  decisions  such  as  funding  drills,  Source  Selection,  and  so  forth. 

Beta  tests  were  completed  and  out-briefed  on  August  27,  2014.  An  implementation  plan  was 
approved  for  future  deployment. 

Pilot  Test  With  NAVAIR  Programs 

Future  work  includes  conducting  four  pilot  test  programs  as  part  of  the  deployment 
strategy.  As  of  the  date  of  this  research  paper,  the  pilots  were  not  yet  complete  and  are  not 
included  herein. 

Findings 

This  section  describes  the  findings  of  qualitative  research  involving  11  NAVAIR 
program  managers  (PMs)  and  16  competency  leads  who  met  the  specified  criteria  for  this 
study  related  to  their  experience  with  acquisition.  The  findings  were  derived  from  27 
interviews  and  an  online  stakeholder  survey.  Risk  drivers  were  codified  into  a  model  based 
on  lessons  learned  from  the  PMs’  experience.  A  prototype  LSI/LCI  directional  tool  was 
developed  and  beta  tested  on  a  cross-section  of  NAVAIR  programs.  This  was  conducted 
over  a  10-month  period.  The  composite  findings  described  in  this  section  include  the 
participants’  characterization  and  Lone  Star  Aerospace  Inc.  experience  and  expertise. 

Beta  Test  Summary 

The  time  invested  by  the  IPTs  who  participated  in  the  beta  tests  was  approximately 
three  days  scheduled  across  four  calendar  days:  a  half-day  introduction,  two  days  of  data 
gathering,  and  a  half  day  of  running  the  model.  The  model  was  quickly  tailored  to  assess  the 
relative  risk  impacts  for  each  LSI  approach.  The  model  quantitatively  demonstrated  relatively 
low  risk  for  government  LSI  approaches  for  one  of  the  PMA-led  efforts  in  which  known 
contributions  to  risk  were  accurately  predicted.  The  model  was  used  in  conversation  with 
other  PMAs  to  quantitatively  demonstrate  PMA  success  and  compare  government 
approaches  to  industry  alternatives.  The  ability  to  execute  multiple  CoAs  was  demonstrated, 
allowing  for  “what-if’  CoA  assessments  prior  to  execution  to  assess  the  relative  risk  of  each 
proposed  approach.  The  structured  process  of  discussing  and  inputting  the  various  options 
also  led  to  better  overall  IPT  understanding  of  the  risks  associated  with  each  approach. 

Observed  Benefits  for  the  Program  IPT 

Programmatic  areas  of  assessment  are  based  on  known  risk  drivers  and  lessons 
learned  from  current  program  managers  and  IPTs.  Potential  LSI  CoAs  for  the  PM  are 
evaluated  from  cost  risk,  schedule  risk,  and  performance  risk  perspectives.  The  PMA  should 
conduct  the  assessments  as  a  group  or  team  in  the  disciplined  manner  with  the  aid  of  the 
LSI/LCI  directional  tool.  It  is  critical  to  have  the  right  people  in  the  room  during  the 
assessment.  Key  SMEs  include  the  PM,  technical  experts,  acquisition  expert,  AIR  4.2 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  103- 


Should  Cost  expert,  IPT  lead,  and  supply  chain  expert.  The  tool  captures  past  IPT 
experience,  perspectives,  and  insight  applied  to  making  future  programmatic  decisions.  This 
creates  rich  evaluations  and  assessments  of  program  areas  that  may  not  be  discussed 
otherwise.  It  provides  a  quantitative  and  systematic  means  of  making  acquisition  strategy 
decisions.  The  artifacts  generated  are  defendable  and  provide  a  consistent  means  of 
communicating  acquisition  strategies  to  leadership  and  stakeholders.  The  key  is  the  learning 
that  occurs  across  the  entire  IPT  as  CoAs  are  run,  iterated,  and  evaluated. 

Observed  Benefits  for  the  Warfare  Center 

It  is  expected  that  once  deployed,  the  aggregated  data  from  different  acquisition 
category  (ACAT)-type  programs  will  deliver  key  characteristics  and  insights  that  lead  to 
programs  choosing  industry  over  the  warfare  center  as  an  LSI.  It  should  identify  gaps  in 
workforce  competencies  and  arm  government  decision-makers  with  actionable  knowledge 
to  set  strategy  to  become  a  competitive  LSI.  Analysis  of  aggregated  data  should  illuminate 
policy  limitations  that  prevent  government-as-LSI  decisions.  It  should  provide  leadership 
with  the  ability  to  make  informed  decisions  on  the  hiring  and  training  of  personnel  and 
support  the  creation  of  data-driven  strategies  for  growth  and  improvement. 

Interpretations,  Conclusions,  and  Recommendations 

The  concept  of  providing  a  useful  tool  to  the  PMA  for  programmatic  decisions  that 
also  allows  NAWCAD  to  aggregate  data  on  NAWC  capabilities,  people,  and  facilities  proved 
to  be  extremely  insightful.  The  model  delivered  LSI  risk  assessments  as  intended,  with 
useful  feedback  to  the  PMA  and  NAWC  in  the  beta  phase.  The  PMA  response  in  the  beta 
phase  was  positive  beyond  intended  LSI  decisions,  and  the  feedback  indicated  value  for 
programmatic  “what-if”  analysis  for  other  PMA  decisions  such  as  funding  drills,  sources 
selection,  and  so  forth. 

Based  on  the  feedback,  an  architecture  change  to  increase  the  skill  set  fidelity  is 
currently  being  implemented  along  with  IRL,  Proprietary  Data,  and  Pass  Through  cost.  This 
will  result  in  Version  1.1  that  is  ready  for  PMA  use  throughout  NAWC.  Version  1.1  will  be 
deployed  to  institutionalize  this  new  capability  within  the  PMA  community.  This  will  provide 
the  PMA  community  with  immediate  access  to  all  “what-if’  programmatic  analyses,  along 
with  the  quantitative  back  up  necessary  to  support  programmatic  decisions. 

The  data  generated  by  this  tool  also  provides  a  basis  for  the  manpower  skill  set 
requirements  across  different  program  types  for  both  the  PMA  and  NAWC.  It  will  help 
NAWC  to  establish  hiring,  training,  and  knowledge  management  goals  to  meet  its 
customers’  requirements.  Organic  integration  capabilities  provided  by  NAWC  will  then  be 
improved  to  better  support  the  PMA.  Overall,  the  best  value  mix  of  government,  OEM,  and 
support  contractors  can  be  achieved. 

It  is  recommended  that  a  Version  1.2  be  developed  that  provides  a  Web-Based 
Programmatic  Decision  Tool.  This  will  provide  the  basis  for  enculturation  across  all  of 
NAVAIR.  The  large  data  set  of  CoA  analysis  will  provide  NAWC  with  the  information  to 
support  different  ACAT-type  programs  more  effectively  in  the  areas  of  design,  development, 
test,  production,  or  sustainment  based  on  what  is  the  best  value  mix  of  government,  OEM, 
and  support  contractors  for  the  warfighter  and  the  taxpayer.  The  research  focused  on  the 
question,  how  can  program  managers  best  determine  their  acquisition  source  solutions  in  a 
timely  and  cost-effective  manner?  Although  the  LCI/LSI  question  was  the  rationale  for  the 
modeling,  the  simulation  tool  additionally  allows  comparison  of  a  wide  range  of  program 
structure  alternatives  through  WBS  tailoring.  It  was  recommended  that  the  tool  has 
application  far  beyond  LSI  decisions. 
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Concluding  Thoughts 

The  identification,  development,  and  implementation  of  the  Programmatic  Decision 
Tool  for  LSI  decisions  followed  a  Dialogue  Decision  Process.  It  provides  the  PMA  and 
NAWC  with  a  tool  necessary  for  adapting  efficiently  and  effectively  in  a  VUCA  environment 
by  facilitating  collaborative  decision-making  with  quantifiable  rationale,  providing  the 
warfighter  and  taxpayer  with  the  best  value  solution  for  LSI  and  other  programmatic 
decisions. 
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